10. HTML Files in Python

HTML Files in Python

Quiz

With your knowledge of HTML file structure, you're going to use Beautiful Soup to extract our desired Audience Score metric and number of audience ratings, along with the movie title like in the video above (so we have something to merge the datasets on later) for each HTML file, then save them in a pandas DataFrame.

The Jupyter Notebook below contains template code that:

• Creates an empty list, df_list , to which dictionaries will be appended. This list of dictionaries will eventually be converted to a pandas DataFrame (this is the most efficient way of building a DataFrame row by row ).
• Loops through each movie's Rotten Tomatoes HTML file in the rt_html folder.
• Opens each HTML file and passes it into a file handle called file .
• Creates a DataFrame called df by converting df_list using the pd.DataFrame constructor .

Your task is to extract the title, audience score, and number of audience ratings in each HTML file so each trio can be appended as a dictionary to df_list .

The Beautiful Soup methods required for this task are:

• find()
• find_all()

There is an excellent tutorial on these methods ( Searching the tree ) in the Beautiful Soup documentation. Please consult that tutorial if you are stuck.

Resources

• rt_html folder
• gathering.ipynb

Solution

Note: At 3:59, "empty character" was said when "empty string" was intended.

More Information

• Beautiful Soup
• Stack Overflow: Beautiful Soup and Unicode Problems
• Stack Overflow: Python: Removing \xa0 from string